Information processing device, information processing method, and computer program

ABSTRACT

An information processing device includes a control unit that has control so that, when detected a status transition of a user including a rising motion or a moving motion of the user during reproduction of a wide-angle image covering the entire field of view of the user, a real-space image acquired by an outward-facing camera provided on a display device that is worn on the head of the user and has a display unit covering the entire field of view is displayed on the display unit in real-time and a 2D image corresponding to a portion of the wide-angle image is displayed on the real-space image.

FIELD

The present disclosure relates to an information processing device, aninformation processing method, and a computer program.

BACKGROUND

In recent years, virtual reality (VR) technologies that, by using adisplay device to be worn on the head or the face of a user, what iscalled a head mount display (HMD), provide the user with virtual videosas if they were real events have been offered.

In the HMD devices, the display screen is located in front of the user'seyes when worn, and there are a transmissive HMD device for whichexternal situation can be seen through the display screen and a shieldedHMD device for which the field of view is blocked by the display screen.In the shielded HMD device, external visual information for the user isblocked, which makes it easier to obtain the sense of immersion in thecontent, and it can be said that it is suitable for providing a virtualreality space. Note that in the case where the transmittance of thedisplay screen can be controlled, both the transmissive type and theshielded type can be implemented with a single HMD device.

Furthermore, in virtual reality technologies, an image obtained bycutting out a portion of a wide-angle image can be presented byfollowing the movement of the user's head. For example, the movement ofthe user's head can be acquired from a gyro sensor or the like and a360-degree video of the entire circumference (what is called a sphericalimage) that follows the movement of the user's head can be made to beexperienced. Moving the display area in the wide-angle image so as tocancel the movement of the head detected by the gyro sensor canimplement a free-viewpoint viewing and a viewpoint movement environment.

In addition, the user who wears a shielded HMD device to view thecontent or to perform various works can concentrate on the relevantcontent or work but is difficult to understand the external situationand was sometimes not aware of external people approaching tocommunicate. The following Patent Literature 1 discloses anotification-state control method that enables a user wearing a shieldedHMD device and an external person to communicate more smoothly.

CITATION LIST Patent Literature

Patent Literature 1: WO2014/156388

SUMMARY Technical Problem

However, when it is needed a small move or work in the real space whilewearing a shielded HMD device and viewing the content, the HMD device isdetached and attached or the transmittance of the display screen isadjusted to change the HMD device to a transmissive HMD device eachtime, and an important scene may be missed, particularly in the case oflive viewing such as sports broadcasting. Even if it is not liveviewing, the user wants to enjoy the content viewing withoutinterruption as much as possible.

Solution to Problem

According to the present disclosure, an information processing device isprovided that includes a control unit configured to have control sothat, when detected a status transition of a user including a risingmotion or a moving motion of the user during reproduction of awide-angle image covering an entire field of view of the user, areal-space image acquired by an outward-facing camera provided on adisplay device that is worn on a head of the user and has a display unitcovering the entire field of view is displayed on the display unit inreal-time and a 2D image corresponding to a portion of the wide-angleimage is displayed on the real-space image.

According to the present disclosure, an information processing method isprovided that includes controlling, by a processor, so that, whendetected a status transition of a user including a rising motion or amoving motion of the user during reproduction of a wide-angle imagecovering an entire field of view of the user, a real-space imageacquired by an outward-facing camera provided on a display device thatis worn on a head of the user and has a display unit covering the entirefield of view is displayed on the display unit in real-time and a 2Dimage corresponding to a portion of the wide-angle image is displayed onthe real-space image.

According to the present disclosure, a computer program is provided thatcauses a computer to function as a control unit configured to havecontrol so that, when detected a status transition of a user including arising motion or a moving motion of the user during reproduction of awide-angle image covering an entire field of view of the user, areal-space image acquired by an outward-facing camera provided on adisplay device that is worn on a head of the user and has a display unitcovering the entire field of view is displayed on the display unit inreal-time and a 2D image corresponding to a portion of the wide-angleimage is displayed on the real-space image.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram explaining the outline of an information processingsystem according to one embodiment of the present disclosure.

FIG. 2 is a block diagram illustrating one example of a configuration ofa display device in the present embodiment.

FIG. 3 is a flowchart illustrating one example of the flow of anoperation process by a first example.

FIG. 4 is a diagram illustrating a display example of a 2D imagerepresenting a portion of a spherical image by the first example.

FIG. 5 is a flowchart illustrating one example of the flow of anoperation process by a second example.

FIG. 6 is a transition diagram illustrating a display example of 2Dimages representing a portion of a spherical image by the secondexample.

FIG. 7 is a transition diagram illustrating another display example of2D images representing a portion of a spherical image by the secondexample.

FIG. 8 is a flowchart illustrating one example of the flow of anoperation process by a third example.

FIG. 9 is a diagram illustrating a display example of a 2D imagerepresenting a portion of a spherical image by the third example.

FIG. 10 is a diagram explaining the display position of a 2D image ofcontent when working in a real space in the present embodiment.

FIG. 11 is a flowchart illustrating one example of the flow of anoperation process by a fourth example.

FIG. 12 is a diagram explaining one example of a user-statusnotification display for other users in a virtual space by a fifthexample.

FIG. 13 is a display example of a 2D image by the fifth example.

FIG. 14 is a diagram illustrating one example of a perforated display ofthe real space in the present embodiment.

FIG. 15 is a diagram explaining a display example of a spherical imagesplit up in a traveling direction in the present embodiment.

DESCRIPTION OF EMBODIMENT

With reference to the accompanying drawings, the following describes indetail a preferred embodiment of the present disclosure. In the presentspecification and the drawings, constituent elements having asubstantially identical functional configuration are denoted byidentical reference signs, so that duplicate description will beomitted.

Furthermore, the description will be given in the following order.

1. Outline of Information Processing System in One Embodiment of PresentDisclosure

2. Configuration Example

3. Operation Process

-   -   3-1. First Example    -   3-2. Second Example    -   3-3. Third Example    -   3-4. Fourth Example    -   3-5. Fifth Example    -   3-6. Sixth Example

4. Others

5. Summary

1. OUTLINE OF INFORMATION PROCESSING SYSTEM IN ONE EMBODIMENT OF PRESENTDISCLOSURE

FIG. 1 is a diagram explaining the outline of an information processingsystem according to one embodiment of the present disclosure. Theinformation processing system in the present embodiment includes, asillustrated in FIG. 1, a display device 10 (information processingdevice) and a content server 20.

The display device 10 is a device having a display unit 150 (FIG. 2)that presents an image of virtual space to a user. For example, thedisplay device 10 is implemented by an HMD device to be worn on the heador the face of the user. The HMD device has an image display unit foreach of the left and right eyes and is configured such that, by usingheadphones in combination, vision and hearing can be controlled. Inaddition, the HMD device can also project different videos to the leftand right eyes, and by displaying images having parallax for the leftand right eyes, can also present a 3D image.

Furthermore, the display device 10 has an inertial measurement unit(IMU) that senses the real space in order to recognize the position andposture of the user, an outward-facing camera, an inward-facing camera,a microphone, and the like.

The display device 10 acquires content of a virtual space from thecontent server 20. The display device 10 generates, from the acquiredcontent, a free-viewpoint image according to the movement of the userand displays it on the display unit 150. The movement of the userincludes the posture of the user and, based on the various sensing datafor which the user was sensed, the display device 10 follows the postureof the user. For example, the display device 10 can, based on thesensing data, perform head tracking and eye-tracking. In addition, theinformation processing system in the present disclosure may furtherinclude a remote controller or a game controller to communicate theintent of the user to the system. In the case where the user is expectedto view virtual space content in a sitting position or standing stillstate, it is assumed that the forward, backward, upward, downward,leftward, and rightward movement in a virtual space can be moved by theoperation of the remote controller or the game controller.

In the present embodiment, it will be described that the generation offree-viewpoint images and the tracking processing are performed by thedisplay device 10, but the present disclosure is not limited thereto,and it may be configured to be performed by an external device.

Furthermore, the content of the virtual space provided to the user isspherical content, free-viewpoint content, game content, or the like.The spherical content is the content of 360-degree video of the entirecircumference (what is called a spherical image), but it may be awide-angle image (for example, a 180-degree image) that covers at leastthe entire field of view of the user. In addition, the free-viewpointcontent is the content that, by using video captured by using aplurality of cameras, generates video when placing a virtual camera atany desired position, so that video viewing can be allowed from anyviewpoint. For example, a person or an object acquired from a pluralityof cameras is 3D modeled and then combined and complemented, therebyallowing viewing from any desired viewpoint including an angle wherethere is no camera. The content is recorded or in real-time.

The term “virtual space” (what is called VR space) used herein refers tothe representation of a real or fictitious environment having rules ofinteraction that is simulated by one or more processors for which theactual user can sense via one or more display devices and/or interactvia one or more user interfaces. In a virtual space, the user may berepresented by an avatar or, without displaying the avatar on thedisplay, a virtual space world from the viewpoint of the avatar may bedisplayed. In the present specification, the viewpoint of the user (orof the avatar) in the virtual space can be regarded as the field of viewof a virtual camera. The “virtual camera” refers to the viewpoint in thevirtual space and is used for calculations to draw a three-dimensionalvirtual space as a two-dimensional image on the display (displaydevice).

BACKGROUND

While viewing the content of a spherical image or at least a wide-angleimage that covers the entire field of view of the user by wearing an HMDdevice in a state of substantially blocking the entire field of view ofthe user, when a small movement or work in the real space is needed suchas going to a bathroom, making coffee, responding to a visit, or thelike, it has been needed that the user detaches and attaches the HMDdevice or operates it such that the real space can be seen each time. Inthis case, in particular, in viewing live such as sports broadcasting orevent broadcasting, an important scene may be missed while leaving theseat and non-displaying the content. In addition, even if it is notviewing live, it is desirable to enjoy the content viewing withoutinterruption as much as possible while being away from the seat also.

Thus, in the present disclosure, offered is a scheme capable ofimproving usability by allowing the user to leave the seat whilecontinuing to view a portion of a wide-angle image during the viewing ofthe wide-angle image covering the entire field of view of the user.

2. CONFIGURATION EXAMPLE

FIG. 2 is a block diagram illustrating one example of a configuration ofthe display device 10 in the present embodiment. As illustrated in FIG.2, the display device 10 in the present embodiment includes a controlunit 100, a communication unit 110, a sensor unit 120, an outward-facingcamera 130, an operation input unit 140, the display unit 150, a speaker160, and a storage unit 170.

2-1. Sensor Unit 120

The sensor unit 120 senses a user status or surrounding situation at acertain period and outputs the sensed information to the control unit100. The sensor unit 120 has a plurality of sensors such as aninward-facing camera 121 that captures the user's eyes when the displaydevice 10 is worn on the head or the face of the user, a microphone 122that collects the user's voice or surrounding sound, an IMU 123 thatsenses the movement of the user, and a direction sensor 124 that detectsthe orientation of the display device 10.

The inward-facing camera 121 is a camera (for example, an infraredsensor having an infrared emitting unit and an infrared imaging unit)that captures the eyes of the user wearing the display device 10. Theinward-facing camera 121 may be provided both for right-eye capturingand left-eye capturing or may be provided only for one of them.

The microphone 122 collects the user's voice and ambient sound (such asenvironmental sounds) and outputs an audio signal to the control unit100.

The IMU 123 is one example of a motion sensor that has a three-axis gyrosensor and a three-axis acceleration sensor and can calculate athree-dimensional angular velocity and acceleration. Note that themotion sensor may be a sensor capable of detecting a total of nine axesby further having a three-axis geomagnetic sensor. Alternatively, themotion sensor may be a sensor of at least one of a gyro sensor and anacceleration sensor.

The direction sensor 124 is a sensor that measures the orientation ofthe display device 10 and is implemented by a geomagnetic sensor, forexample.

2-2. Control Unit 100

The control unit 100 functions as an arithmetic processing unit and acontrol device and controls the overall operation of the display device10 in accordance with various computer programs. The control unit 100 isimplemented by an electronic circuit such as a central processing unit(CPU) and a microprocessor, for example. The control unit 100 may alsoinclude a read only memory (ROM) that stores computer programs,calculation parameters, and the like to be used and a random accessmemory (RAM) that temporarily stores parameters and the like that changeas appropriate.

In addition, the control unit 100 in the present embodiment alsofunctions as a posture-information acquisition unit 101, amovement-information acquisition unit 102, a line-of-sight informationacquisition unit 103, a content-important area identification unit 104,and a drawing processing unit 105.

The posture-information acquisition unit 101 acquires (calculates),based on the sensing data acquired from the sensor unit 120, the postureinformation (including the head posture) on the user. For example, theposture-information acquisition unit 101 can calculate, based on thesensing data of the IMU 123 and the direction sensor 124, the userposture including the head posture of the user. This also makes itpossible to grasp a transition state of the user's body such as whetherthe user has sat down or stood up.

The movement-information acquisition unit 102 acquires (calculates),based on the sensing data acquired from the sensor unit 120, theinformation concerning the actual movement of the user in the realspace. For example, the movement-information acquisition unit 102acquires, based on the sensing data of the IMU 123 and the directionsensor 124, the movement information including walking of the user, thedirection of travel, and the like.

The line-of-sight information acquisition unit 103 acquires(calculates), based on the sensing data acquired from the sensor unit120, line-of-sight information on the user. For example, theline-of-sight information acquisition unit 103 calculates, based on thesensing data of the inward-facing camera 121, the line-of-sightdirection of the user and a gazing point (line-of-sight position). Inthis case, the inward-facing camera 121 is used as one example of aline-of-sight detection sensor for acquiring (calculating) the user'sline-of-sight information, but the line-of-sight detection sensor is notlimited thereto and may be a myoelectric sensor that detects themovement of muscles around the user's eyes or an electroencephalogramsensor. Furthermore, the line-of-sight information acquisition unit 103may, by using the head posture (orientation of the head) acquired by theposture-information acquisition unit 101, estimate the line-of-sightdirection in a pseudo manner.

The content-important area identification unit 104 has a function toidentify an important area in the content that the user is viewing. Thecriteria for determining the important area may be set in advanceaccording to the content. For example, in the case of live video offootball, an area containing at least the ball and the player closest tothe ball is identified as an important area.

The drawing processing unit 105 performs generation and display controlof an image to be displayed on the display unit 150. For example, as inthe foregoing, the drawing processing unit 105 generates, from thecontent (for example, a spherical image that is the video of virtualspace) acquired from the content server 20, a free-viewpoint imageaccording to the operating input by the motion of the user or thecontroller and displays it on the display unit 150.

In addition, the drawing processing unit 105 in the present embodimenthas control so that the real-space image acquired by the outward-facingcamera 130 provided in the display device 10 is displayed on the displayunit 150, and the outside world can be seen even in a state of wearing ashielded HMD device (what is called video see-through). This enables theuser to perform a safe moving or work in the real space, in a state ofwearing the shielded HMD device.

In this case, when the user rises and leaves the seat (detection of astatus transition of the user including a rising motion or moving motionof the user) while viewing a wide-angle image covering at least theentire field of view of the user such as a spherical image, the drawingprocessing unit 105 has control so that the real-space image isdisplayed in real-time and a 2D image representing a portion of thewide-angle image that has been viewing is displayed on the real-spaceimage. This allows the user to continue viewing at least a portion ofthe wide-angle image while leaving the seat and moving or working in thereal space. For example, by displaying the important area of the contentas a portion of the wide-angle image, it is possible to prevent missingan important scene while being away from the seat. Furthermore, thedrawing processing unit 105 may include the area (gazing area) at whichthe user was looking immediately before leaving the seat in a portion ofthe wide-angle image to be displayed. Furthermore, the drawingprocessing unit 105 may place the 2D image representing a portion of thewide-angle image at a certain location in a local coordinate system withreference to the display device 10.

2-3. Communication Unit 110

The communication unit 110 connects to the content server 20 by wire orwirelessly to transmit and receive data. The communication unit 110connects to and communicates with the content server 20 by awired/wireless local area network (LAN), Wi-Fi (registered trademark),Bluetooth (registered trademark), or the like.

2-4. Outward-Facing Camera 130

The outward-facing camera 130 images the real space and outputs acaptured image (real-space image) to the control unit 100. A pluralityof outward-facing cameras 130 may be provided. For example, by a stereocamera on which a plurality of outward-facing cameras 130 are provided,a right-eye image and a left-eye image can be acquired.

2-5. Operation Input Unit 140

The operation input unit 140 detects the operating input of the user tothe display device 10 and outputs the operating input information to thecontrol unit 100. The operation input unit 140 may be a touch panel,buttons, switches, levers, and the like, for example. Furthermore, theoperating input of the user to the display device 10 is not limited tothe physical operation by the buttons and the like but may be gestureinput or voice input. The user can also use a controller separate fromthe display device 10 to perform operating input.

2-6. Display Unit 150

The display unit 150 includes, when the display device 10 is configuredas an HMD device, the left and right screens fixed to correspond to theleft and right eyes, respectively, of the user and displays the left-eyeimage and the right-eye image. Furthermore, the display unit 150 isprovided so as to cover at least the entire field of view of the user.Furthermore, the screen of the display unit 150 may be a display panelsuch as a liquid crystal display (LCD) or an organic electroluminescence (EL) display.

2-7. Speaker 160

The speaker 160 is configured as headphones to be worn on the user'shead when the display device 10 is configured as an HMD device andreproduces an audio signal. Furthermore, the speaker 160 is not limitedto the headphone type and may be configured as earphones or boneconduction speakers.

2-8. Storage Unit 170

The storage unit 170 is implemented by a read only memory (ROM) thatstores computer programs, calculation parameters, and the like to beused for the processing of the control unit 100 and a random accessmemory (RAM) that temporarily stores parameters and the like that changeas appropriate. In addition, in the storage unit 170, the content(spherical image and the like) of virtual space acquired from thecontent server 20 may be stored.

As in the foregoing, the configuration of the display device 10 has beendescribed specifically, but the configuration of the display device 10by the present disclosure is not limited to the example illustrated inFIG. 2. For example, at least a part of the functional configuration ofthe control unit 100 may be provided in an external device (for example,an information processing device that connects to and communicates withthe display device 10, a server on a network, or the like).

In addition, when the content is not that delivered from the contentserver 20 in real-time such as live video, the content can bereproduced, by storing the content in the storage unit 170 in advance,even in a state of not being connected to the network.

3. OPERATION PROCESS

Next, the flow of display control processing according to the statetransition of the user during the reproduction of a wide-angle imagecovering the entire field of view of the user in the present embodimentwill be described specifically with reference to the drawings. In thiscase, as one example of a wide-angle image, a spherical image is used.

3-1. First Example

FIG. 3 is a flowchart illustrating the flow of an operation process by afirst example. As illustrated in FIG. 3, while the user wearing thedisplay device 10 is viewing the spherical image in a sitting position(Step S103), the display device 10 continuously detects, by theinward-facing camera 121, the line of sight of the user (Step S106).

Then, the display device 10 detects, based on the sensing data sensedfrom the IMU 123, a rising motion (standing position) of the user by theposture-information acquisition unit 101 (Step S109). When the posturesimply moves while in the sitting position (such as leaning forward orlooking around), it is not detected as a standing position, and thedisplay device 10 can move the viewpoint of the user in the virtualspace according to the change in the user's posture.

Then, if a standing position of the user is detected (Yes at Step S109),the display device 10 assumes that the user leaves the seat, andperforms the processing of turning on the outward-facing camera 130(Step S112), the identifying of the gazing point of the user in thespherical image immediately before the standing position detection (StepS115), the identifying of an important area in the content (sphericalimage) (Step S118), and the disabling of operation on the virtual spacesuch as free-viewpoint operation (Step S127). Those processes may beprocessed in parallel.

The outward-facing camera 130 that acquires a real-space image isbasically turned off while the user is viewing a spherical image inorder to reduce power consumption, but because moving or work in a realspace is expected when the user rises, the outward-facing camera 130 iscontrolled to be on and the real-space image is acquired.

Subsequently, the display device 10 generates a 2D image including theidentified gazing area and the important area from the spherical imageby the drawing processing unit 105.

Then, the display device 10 displays the real-space image on the displayunit 150 and, on the real-space image, displays a portion of thespherical image, specifically, a 2D image including the important areaand further a 2D image including the gazing area at which the user wasgazing immediately before (Step S124). FIG. 4 is a diagram illustratinga display example in the display unit 150 of the display device 10 inthe first example. As illustrated on the left side of FIG. 4, while theuser is viewing the spherical image using the display device 10 in asitting state, when the user rises, as illustrated on the right side ofFIG. 4, a real-space image 400 is displayed so as to put it in a videosee-through state, and a 2D image 500 including a gazing area 310 atwhich the user was gazing immediately before and an important area 320(for example, a football and a chasing player) of the present content isdisplayed on the real-space image 400 in a superimposed manner.

As a result, at least a portion of the content can be viewedcontinuously without disturbing the moving or the work in the real spaceand, in particular, by displaying the 2D image including at least theimportant area 320 of the content, missing an important scene such asthe moment of a goal, for example, can be prevented.

The display position of the 2D image 500 is not specifically limited,and it may be displayed at a certain position (for example, below thefield of view) in the local coordinate system with reference to thedisplay device 10, for example. When the user starts to walk, it may beplaced at a position not interfering with the movement of the user.

The shape of the 2D image 500 may be a quadrangle as illustrated in FIG.4 or may be a circular shape, an elliptical shape, or the like. Thedisplay device 10 may further make the layer of the background image bethe real-space image, and by continuously reducing the area of thespherical image and switching it to the 2D image 500, may display it onthe real-space image in a superimposed manner.

The display device 10 further controls the display angle of view so thatan important portion of the spherical image is always visible in the 2Dimage 500.

3-2. Second Example

In the above-described example, the case where the 2D image 500 havingan angle of view including the last gazing area of the user and theimportant area is displayed has been described, but when the gazing areaand the important area are too far apart or the like, the video may betoo reduced and be hard to see, when the angle of view is made toinclude both areas. Thus, as a second example, the display control thattransitions the angle of view of a 2D image to an important area so thatthe user does not lose sight of the moving destination of the displayangle of view will be described with reference to FIG. 5 to FIG. 7.

FIG. 5 is a flowchart illustrating the flow of an operation process bythe second example. As illustrated in FIG. 5, the display device 10first identifies a gazing point of the user immediately before detectinga standing position (Step S203) and also identifies an important area ofthe content (spherical image) (Step S206).

Next, the display device 10 calculates the distance between the gazingarea and the important area (Step S209).

Then, the display device 10 determines the magnification of theimportant area (Step S212). The magnification of the important area ispredetermined. For example, in the case of football broadcasting, themagnification at which the ball is in a certain size (recognizable size)may be set.

Next, the display device 10 determines whether both the gazing area andthe important area are within the presentation range of the 2D image(that is, the display size of the 2D image, angle of view) (Step S215).Because the display size of the 2D image is also predetermined basically(as an exception, because it may be controlled to be small when the useris working and the like in the real space, so as not to disturb the workarea), the display device 10 may determine based on whether the distancebetween the gazing area and the important area exceeds a certain value,for example. Furthermore, when the image is reduced and displayed, itmay fit within the presentation range of the 2D image, but if it is toosmall, it is difficult to see, and thus, as in the foregoing, byconsidering the predetermined magnification, it may be determinedwhether both the gazing area and the important area are within a certainangle of view.

Then, if both are included (Yes at Step S215), the drawing processingunit 105 of the display device 10 generates a 2D image that includesboth (Step S218).

On the other hand, if not both are included (No at Step S215), thedrawing processing unit 105 of the display device 10 generates 2D imagesthat transition while changing the display magnification from the gazingarea to the important area (Step S221). A specific display example willbe described later with reference to FIG. 7. The drawing processing unit105 of the display device 10 may generate 2D images that focus (magnify)from a wide range (reduced image) including both the gazing area and theimportant area to the important area. Such an example will be describedlater with reference to FIG. 6.

Then, the display device 10 controls so that, on the display unit 150, a2D image of the content (spherical image) is displayed with thebackground of the real-space image (Step S224).

Specific Display Example

Specific display examples of 2D images in the second example will bedescribed with reference to FIG. 6 and FIG. 7.

FIG. 6 is a transition diagram illustrating a display example of 2Dimages representing a portion of a spherical image by the secondexample. For example, as illustrated in the upper left of FIG. 6, whenthe gazing area 310 and the important area 320 are apart and are notwithin the presentation range of the 2D image, the display device 10 mayfirst, as illustrated in the upper right of FIG. 6, present a 2D image500 a including both the gazing area 310 and the important area 320 byreducing the image, and then, as illustrated in the lower left of FIG.6, present a 2D image 500 b that focused on the important area 320, andfinally, as illustrated in the lower right of FIG. 6, display a 2D image500 c in which the important area 320 was enlarged to a certainmagnification.

In addition, as illustrated in the upper left of FIG. 7, similarly, whenthe gazing area 310 and the important area 320 are apart and are notwithin the presentation range of the 2D image, the display device 10 mayfirst, as illustrated in the upper right of FIG. 7, present a 2D image500 d including the gazing area 310, and then, as illustrated in thelower left of FIG. 7, present a 2D image 500 e in which the displayangle of view transitions toward the important area 320, and finally, asillustrated in the lower right of FIG. 7, display a 2D image 500 f thatincludes the important area 320 and was enlarged to a certainmagnification.

By transitioning from the gazing area to the important area in this way,the user can be prevented from losing sight of the moving destination ofthe display angle of view and not knowing where it has transitioned.

Furthermore, the display device 10 may, in order to clarify thetransition destination of the display angle of view, highlight theimportant area (for example, illuminating it, surrounding it withblinking lines, or displaying an arrow) in advance, and then transition.

Furthermore, the display device 10 may, when the gazing area isdifferent from the important area portion, annotate (may highlight aswith the foregoing).

Furthermore, the display device 10 may display the gazing area for aspecific period of time and then transition from there to the importantarea (at that time, a countdown display and the like may be used).

3-3. Third Example

Subsequently, the control of the display position of the 2D image 500when the user rises and further starts moving will be described withreference to FIG. 8 and FIG. 9.

FIG. 8 is a flowchart illustrating the flow of an operation process by athird example. As illustrated in FIG. 8, the display device 10 firstdetects, after being in a standing position state (in a state where areal-space image and a 2D image are presented on the display unit 150)(Step S303), the movement of the user's body (Step S306). In this case,when actually started walking and the like, but not the posture changein a degree of leaning forward, is performed, it is detected as themovement of the body. For example, the movement-information acquisitionunit 102 of the display device 10 can detect, based on the sensing datafrom the IMU 123, the direction sensor 124, and the communication unit110, the movement of the user's body.

Next, the display device 10 determines whether the 2D image can bedisplayed at a position that does not disturb the movement (Step S309).The position that does not disturb the movement is assumed to be, in areal-space image, a place avoiding the traveling direction of or thefeet of the user, a wall, a ceiling, and the like. For example, it canbe determined that, when it can be estimated that the travelingdirection of the user is straight from the shape of the corridor and thelike, it can be placed on the left or the right side of the corridor,and that, when there are many objects on a passage and the user has towalk carefully, it can be placed upward.

Next, when determined that it can be displayed at a position that doesnot disturb the movement (Yes at Step S309), the display device 10controls so as to display the 2D image at a position that does notdisturb the movement of the user (Step S312). Herein, FIG. 9 illustratesa display example of a 2D image representing a portion of a sphericalimage by the third example. As illustrated in FIG. 9, for example, whenthe user is walking toward the kitchen located to the right in thereal-space image 400, the display device 10 displays the 2D image 500 tothe left.

On the other hand, when determined that it is not possible to bedisplayed at a position that does not disturb the movement (No at StepS309), the display device 10 turns the 2D image into non-display andpauses the content viewing, and secures the field of view in the movingdirection (Step S315). In this case, when the user is in a sittingposition and is ready to view again, the chased reproduction or thechased high-speed reproduction of the content, the digest reproductionduring stopping, and the like may be performed.

Line-of-Sight Use

The display device 10 in the present embodiment may, by using the lineof sight of the user, control the reproduction of the content (sphericalimage).

For example, in the case of non-real-time content such as live video,the display device 10 may continue the reproduction when the user's lineof sight is within the content (for example, within a 2D imagerepresenting a portion of the content superimposed on the real-spaceimage) for a specific period of time or longer, and when the line ofsight is off for a specific period of time or longer, pause thereproduction. When controlling the reproduction and the stopping by theline-of-sight state, at the time of resuming, the reproduction may bestarted in duplicate from the portion for a specific period of timebefore stopping.

Furthermore, while the certainty factor of the IMU is low (when theestimation was not successful), when the gazing point of the user iscontinuously on the viewing content (object in the content), the displaydevice 10 may determine that the user is simply moving the posture, nota standing position, and prioritize the content viewing. Conversely,when the gazing point is not on the object in the content or is moved tothe periphery, it may be determined to be a standing position.

Furthermore, the display device 10 may personalize the important areaaccording to the line of sight. It is useful when the important area maydiffer from person to person. Specifically, when the gazing pointdistribution in the content viewing space is concentrated at a specificposition within a specific period of time, the display device 10 canpresume the relevant portion as an important area for the user andpersonalize the important area. Then, for example, while the user's lineof sight is in the real-space image, at least the personalized importantarea of the content is stored (recorded), and when returned to thecontent viewing, the chased reproduction or the chased high-speedreproduction of the relevant important area, or the digest reproductionduring stopping may be performed.

Display of 2D Image in Working in Real Space

The display device 10 may, when the moving speed is less than or equalto a certain value and the gazing point is not directed to the content(2D image), assume being in work (making coffee, going to a bathroom,responding to delivery service, or the like) in a real space. In thiscase, the display device 10 determines, according to the movement of theline of sight, the presentation position of the content (2D image). Forexample, as illustrated in FIG. 10, the locus of a gazing point L isanalyzed when the user is working at hand or at his/her feet and aposition that does not disturb the work is identified, and then theplacement of a 2D image 570 of the content is adjusted. For example, inthe vicinity of the line-of-sight position in working, a position wherethere is no object, a position where the body such as a user's hand isnot detected, and the like can be assumed.

Furthermore, the display device 10 may, by analyzing the real-spaceimage and detecting in advance what work the user is trying to do,control the placement of the 2D image. For example, when it is predictedthat coffee is made, the places where sugar, milk, and the like arepresent are also excluded from the 2D image placement possible area inadvance.

The display device 10 can further perform reproduction control accordingto the motion range of the user. Specifically, when the work area of theuser extends beyond a specific range in the real space such as thepresence of changes in the body direction greater than or equal to aspecific frequency (work in a wide range such as drying laundry, movingobjects, or the like), the reproduction of the 2D image of the contentmay be paused in order not to disturb the work, and be resumed when theuser is ready to view such as sitting down again.

Furthermore, the display device 10 may pause the content (2D image) whendetected that the user is interacting with others in the real space.

Display Example of 2D Image

The display device 10 may analyze in advance the content that is notreal-time video, and present the relevant information to the user whenthe time of excitement is near, or impart an effect that makes thedisplay angle of view difficult to shrink or an effect that is differentfrom normal when shrinking, in changing the spherical image to a 2Dimage when the time of excitement is near. Even if the real-timecontent, when the excitement is expected, the relevant information maybe presented to the user.

Furthermore, the display device 10 may control the reproduction of thecontent according to the surrounding environment. For example, thedisplay device 10 may, when the real space is a bright place, increasethe brightness of the content (2D image) and present it in a large size.In addition, the display device 10 may, in a noisy place, reduce thesound of the 2D-imaged content not so much. In addition, the 2D image ofthe content may be displayed at a place where the color of the realspace is different from the hue of the content.

Furthermore, when the important area cannot be identified, the displaydevice 10 may default to the display control in which the display angleof view is reduced toward the center of the spherical image (forexample, the center of the user's line of sight) to form a 2D image.Alternatively, the place where the character is present in the sphericalimage may be regarded as an important area.

In addition, the user may be allowed to arbitrarily set the mode inwhich the 2D-image display can be viewed continuously.

The shape of the 2D image is not limited to a uniform square or circleand may be scaled to a shape suitable for the content. The 2D image maybe a plurality of videos (for example, a score display screen in thecontent of football broadcasting, a play display screen for around theball, and the like).

Furthermore, when switching the display by detecting a standing positionor movement of the user, the display device 10 can also shift anintermediate state of transition of the display screen by animation soas to prevent the screen from switching abruptly. Furthermore, when thetransition from a sitting position to a standing position is detected,the 2D image may be displayed at the same specific position (forexample, the center of the field of view) each time, and then be movedto an appropriate position (the user can be made not to miss the imagethat has been changed from the spherical image to the 2D image).

Furthermore, it is expected that fluttering may occur due to theinfluence of the detection section of the sensing, but meanwhile, it isdesirable to switch without latency as much as possible. For thisreason, the display device 10 may start switching the display whendetected a standing position, but not complete the transition for aspecific period of time, and then complete the transition of switchingwhen the certainty factor of recognizing the moving motion is greaterthan or equal to a specific value, for example. In addition, when theuser's line of sight can be used, the transition in switching may bestarted at the time the line of sight was shifted to the real-spaceimage (of the background).

3-4. Fourth Example

Subsequently, an operation process of returning to the viewing of thespherical image will be described with reference to FIG. 11. FIG. 11 isa flowchart illustrating the flow of an operation process by a fourthexample.

As illustrated in FIG. 11, the display device 10 first determines, whendetected stopping or a sitting position (Yes at Step S406) after beingin a moving state (a state in which a real-space image and a 2D imageare presented) (Step S403), whether the line of sight (gazing point) isin the content (2D image displayed in superimposition on the real-spaceimage) (Step S406).

Next, if the line of sight (gazing point) is in the content (Yes at StepS409), the display device 10 turns off the outward-facing camera 130(Step S412) and has control of returning to viewing the spherical image(Step S415).

On the other hand, if the line of sight (gazing point) is not in thecontent (No at Step S409), the display device 10 assumes that the workis being performed in the real space, and has control so that the 2Dimage of the content is displayed in a position that does not disturbthe work and at which the line of sight can follow (Step S418).

3-5. Fifth Example

Furthermore, the display device 10 of the present embodiment is not onlylimited to the display control of the content but also can have controlconcerning the sound. For example, when the user is moving or working inthe real space, because there is a need to hear the ambient sound, thedisplay device 10 may continue the reproduction by reducing the volumeof the content (2D image). Furthermore, when the AR representation ofthe sound is performed, this is stopped. When the reproduction of thecontent that is not real-time is stopped, the display device 10 alsostops the sound. Conversely, when stopping the reproduction of thereal-time content, only the sound may be reproduced in order to preventoverlooking.

Furthermore, while there is an important event or excitement, when theuser is not watching the content (2D image), the display device 10 can,by warning with the sound and highlighting the excitement, achievepreventing overlooking.

The display device 10 may, when experiencing a virtual space such asviewing a spherical image, cancel the ambient sound because it isdifficult to concentrate on the content when the ambient sound is heard.Alternatively, the display device 10 may adjust the speaker 160 so thatthe ambient volume is relatively small with respect to the volume of thecontent.

In this case, it becomes difficult for the user to notice the ambientsound, but it may be configured not to cancel the ambient sound that isa certain trigger (for example, a doorbell, a family call, and thelike), for example. The general sounds can be registered in DB ascancellation-exclusion sounds, and the other sounds can be registered inadvance by the user.

Furthermore, when the user rises and spontaneously moves, at the timingof turning on the outward-facing camera 130, the cancellation of theambient sound may be stopped.

3-6. Sixth Example

Next, an example other than the viewing type content will be described.For example, in the case of communication-type content in which aplurality of people play a game in a virtual space, because it may bebaffling when the other user (avatar and the like) suddenly stops movingor stops responding in the game, it may be configured to switch thecontent display and notify the communication partners of status.

For example, when a standing position was detected in the case of asitting game, or when a movement was detected in the case of a standinggame, the display device 10 switches the display so as to display thereal-space image on the background, turns a portion of the sphericalimage of the game screen into a 2D image and superimposes it on thereal-space image, and disables the operation in the virtual space. As aresult, when viewed from other users playing the game together in thevirtual space, it is seen such that the avatar and the like of therelevant user suddenly stops moving or stops responding in the game.Thus, at this time, the display device 10 notifies the content server 20of status (for example, “started moving”, “moving”, “suspended”, “awayfrom seat”, and the like), so that, as illustrated in FIG. 12, in a gamescreen 600 (spherical image) the other users are viewing, a statusnotification image 602 such as “moving” is displayed on an avatar 601 ofthe relevant user. This allows the other users to intuitively understandthat the relevant user is in the middle of moving or working in the realspace.

Note that even during the 2D image display, it may be possible torespond to appeals from the virtual space. For example, the displaydevice 10 may emphasize the appealing sound to the user.

Furthermore, even when the game is suspended due to a standing positionor moving, the 2D image of the game that progresses in real-time isdisplayed around the gazing area and the like so as not to disturb themoving or working, so that the progress of the game can be grasped tosome extent. Furthermore, when the game is resumed, high-speedreproduction (digest reproduction) may be performed. Furthermore, whenthe game is being suspended due to a standing position or moving, onlythe sound of the virtual space may be kept in a reproduction state.

In addition, even when switched to the display of the 2D image (whenleaving the seat), operating the game may be allowed. FIG. 13 is adisplay example of a 2D image 620 by the present example. Because thecontroller is difficult to use while moving, the display device 10 mayenable the operating input by touch gestures to the 2D image 620illustrated in FIG. 13, or voice. However, because the erroneousoperation to the 2D image 620 is likely to occur when the moving orworking in the real space is involved, by combining voice recognitionand gesture, activation may be performed by voice, or operationconfirmation may be performed by voice. In addition, the line of sightand the gesture may be combined so that the operation can be performedonly where the user is looking.

Furthermore, in the case of virtual space content (such as games) thatassumes physical movement due to stepping and the like of the user, thedisplay device 10 may regard a mere stepping motion or posture change asan operation of the content, and regard actual walking for a specificdistance or more (for example, when acceleration in a specific directionis detected for a specific level or more) as detection of walking(moving) and secure the field of view needed for the moving (display thereal-space image).

The display device 10 may, when moving out of a specific area set inadvance such as leaving the room where viewing is taken place, suspendthe content viewing and display the real-space image.

Furthermore, the display device 10 may, from the viewpoint of avoidingdanger, always present the environment of the surrounding real space.For example, the display device 10 may make the places, which are likelyto cause danger (stepping, bumping, and the like) such as below thefeet, below the left and the right, and the like, always kept visible.Furthermore, the display device 10 may present the surroundingreal-space environment, depending on the situation (dangerous conditionsand the like).

4. OTHERS

The switching of the display according to the detection of a standingposition or moving in the present embodiment is not limited to each ofthe above-described examples. The following describes somemodifications.

For example, when the moving or working in the real space does notfrequently occur, or when the moving is only within a specific range,the display device 10 may make the spherical image translucent anddisplay the entire real-space image to be visible. At this time, thedisplay device 10 may increase the transparency of the spherical imagewhen there are many movements of the gazing point to objects in the realspace, and conversely, when there are many gazing points in thespherical image, may reduce the transparency of the spherical image. Thecontent is kept reproduced, and when the user stops or sits down, thetranslucency is canceled.

Furthermore, the display device 10 may cut out a portion of thespherical image so that the real space is visible. FIG. 14 is a diagramillustrating one example of a perforated display of the real space inthe present embodiment. As illustrated in FIG. 14, on a portion of aspherical image 300, the real-space image 400 is displayed so that itcan be viewed. Where to make the perforation and the size of theperforation (display range of the real-space image) may be determined,for example, according to the distribution of the gazing point of theuser's line of sight. For example, when the real space is being gazedat, the display device 10 may expand the display range of the real-spaceimage.

Furthermore, the display device 10 may display the real-space image inthe moving direction and, in the surrounding portions other than themoving direction, continue the reproduction of the content. Then, whenthe user stops or sits down, the perforated portion in the content thathas been opened for securing the field of view in the real space isfilled, and the reproduction of the spherical image is resumed.

Furthermore, the display device 10 may always fix the display positionof the 2D image of the content such as below the front.

Furthermore, the display device 10 may, as illustrated in FIG. 15,perform display control so that the spherical image appears to be splitup in the traveling direction of the user. In the example illustrated inFIG. 15, the image 500 (spherical image) is split up in the travelingdirection of the user and is displayed on the left and the rightrespectively (2D images 500 g, 500 h).

As in the foregoing, in the present embodiment, it is possible tocontinue viewing at least a portion of the spherical image in thevirtual space while securing the field of view in the real space.

5. SUMMARY

As just described, a preferred embodiment of the present disclosure hasbeen described in detail with reference to the accompanying drawings,but the present technology is not limited to such examples. It is clearthat anyone with ordinary knowledge in the technical field of thepresent disclosure may conceive various alterations or modificationswithin the scope of the technical ideas set force in the claims, and itis naturally understood that they belong to the technical scope of thepresent disclosure.

For example, it is also possible to create a computer program thatexerts the functions of the display device 10 and the content server 20on the hardware such as a CPU, a ROM, a RAM, and the like that are builtinto the display device 10 (information processing device) and thecontent server 20. In addition, a computer-readable storage medium inwhich the relevant computer program is stored is also provided.

The effects described herein are merely explanatory or exemplary but arenot limited. That is, the technology according to the present disclosuremay exhibit other effects apparent to those skilled in the art from thedescription herein, in addition to the above-described effects or inplace of the above-described effects.

Note that the present technology can also take the followingconfigurations.

(1)

An information processing device comprising a control unit configured tohave control so that, when detected a status transition of a userincluding a rising motion or a moving motion of the user duringreproduction of a wide-angle image covering an entire field of view ofthe user, a real-space image acquired by an outward-facing cameraprovided on a display device that is worn on a head of the user and hasa display unit covering the entire field of view is displayed on thedisplay unit in real-time and a 2D image corresponding to a portion ofthe wide-angle image is displayed on the real-space image.

(2)

The information processing device according to (1), wherein the controlunit generates, as a 2D image, an image of a portion including a certainimportant area in the wide-angle image.

(3)

The information processing device according to (2), wherein the controlunit generates, as the 2D image, an image of a portion further includinga gazing area based on line-of-sight information on the user in thewide-angle image immediately before detecting the status transition.

(4)

The information processing device according to (1), wherein the controlunit generates 2D images that transition from an image of a portionincluding a gazing area based on line-of-sight information on the userand a certain important area in the wide-angle image immediately beforedetecting the status transition to an enlarged image of the importantarea.

(5)

The information processing device according to (1), wherein the controlunit generates 2D images that transition from a gazing area based online-of-sight information on the user in the wide-angle imageimmediately before detecting the status transition to a certainimportant area.

(6)

The information processing device according to any one of (1) to (5),wherein the control unit controls a display position of the 2D imageaccording to a moving direction when detected a moving motion of theuser.

(7)

The information processing device according to any one of (1) to (6),wherein the control unit controls a power supply of the outward-facingcamera to turn on when detected the status transition.

(8)

The information processing device according to any one of (1) to (7),wherein the control unit switches from a display of the 2D image to adisplay of a wide-angle image covering the entire field of view of theuser on the basis of line-of-sight information on the user when detecteda movement stop motion or a sitting position of the user.

(9)

An information processing method comprising controlling, by a processor,so that, when detected a status transition of a user including a risingmotion or a moving motion of the user during reproduction of awide-angle image covering an entire field of view of the user, areal-space image acquired by an outward-facing camera provided on adisplay device that is worn on a head of the user and has a display unitcovering the entire field of view is displayed on the display unit inreal-time and a 2D image corresponding to a portion of the wide-angleimage is displayed on the real-space image.

(10)

A computer program that causes a computer to function as a control unitconfigured to have control so that, when detected a status transition ofa user including a rising motion or a moving motion of the user duringreproduction of a wide-angle image covering an entire field of view ofthe user, a real-space image acquired by an outward-facing cameraprovided on a display device that is worn on a head of the user and hasa display unit covering the entire field of view is displayed on thedisplay unit in real-time and a 2D image corresponding to a portion ofthe wide-angle image is displayed on the real-space image.

REFERENCE SIGNS LIST

10 DISPLAY DEVICE

20 CONTENT SERVER

100 CONTROL UNIT

101 POSTURE-INFORMATION ACQUISITION UNIT

102 MOVEMENT-INFORMATION ACQUISITION UNIT

103 LINE-OF-SIGHT INFORMATION ACQUISITION UNIT

104 CONTENT-IMPORTANT AREA IDENTIFICATION UNIT

105 DRAWING PROCESSING UNIT

110 COMMUNICATION UNIT

120 SENSOR UNIT

121 INWARD-FACING CAMERA

122 MICROPHONE

124 DIRECTION SENSOR

130 OUTWARD-FACING CAMERA

140 OPERATION INPUT UNIT

150 DISPLAY UNIT

160 SPEAKER

170 STORAGE UNIT

1. An information processing device comprising a control unit configuredto have control so that, when detected a status transition of a userincluding a rising motion or a moving motion of the user duringreproduction of a wide-angle image covering an entire field of view ofthe user, a real-space image acquired by an outward-facing cameraprovided on a display device that is worn on a head of the user and hasa display unit covering the entire field of view is displayed on thedisplay unit in real-time and a 2D image corresponding to a portion ofthe wide-angle image is displayed on the real-space image.
 2. Theinformation processing device according to claim 1, wherein the controlunit generates, as a 2D image, an image of a portion including a certainimportant area in the wide-angle image.
 3. The information processingdevice according to claim 2, wherein the control unit generates, as the2D image, an image of a portion further including a gazing area based online-of-sight information on the user in the wide-angle imageimmediately before detecting the status transition.
 4. The informationprocessing device according to claim 1, wherein the control unitgenerates 2D images that transition from an image of a portion includinga gazing area based on line-of-sight information on the user and acertain important area in the wide-angle image immediately beforedetecting the status transition to an enlarged image of the importantarea.
 5. The information processing device according to claim 1, whereinthe control unit generates 2D images that transition from a gazing areabased on line-of-sight information on the user in the wide-angle imageimmediately before detecting the status transition to a certainimportant area.
 6. The information processing device according to claim1, wherein the control unit controls a display position of the 2D imageaccording to a moving direction when detected a moving motion of theuser.
 7. The information processing device according to claim 1, whereinthe control unit controls a power supply of the outward-facing camera toturn on when detected the status transition.
 8. The informationprocessing device according to claim 1, wherein the control unitswitches from a display of the 2D image to a display of a wide-angleimage covering the entire field of view of the user on the basis ofline-of-sight information on the user when detected a movement stopmotion or a sitting position of the user.
 9. An information processingmethod comprising controlling, by a processor, so that, when detected astatus transition of a user including a rising motion or a moving motionof the user during reproduction of a wide-angle image covering an entirefield of view of the user, a real-space image acquired by anoutward-facing camera provided on a display device that is worn on ahead of the user and has a display unit covering the entire field ofview is displayed on the display unit in real-time and a 2D imagecorresponding to a portion of the wide-angle image is displayed on thereal-space image.
 10. A computer program that causes a computer tofunction as a control unit configured to have control so that, whendetected a status transition of a user including a rising motion or amoving motion of the user during reproduction of a wide-angle imagecovering an entire field of view of the user, a real-space imageacquired by an outward-facing camera provided on a display device thatis worn on a head of the user and has a display unit covering the entirefield of view is displayed on the display unit in real-time and a 2Dimage corresponding to a portion of the wide-angle image is displayed onthe real-space image.